COMBINATORIAL INVARIANCE OF RELATIVE KAZHDAN-LUSZTIG 
POLYNOMIALS IN THE HERMITIAN SYMMETRIC CASE 



W. ANDREW PRUETT 

Abstract. We develop a marking system for an analog of Hasse diagrams of intervals [u,v] with 
u < v in a Hermitian symmetric pair W/Wj, and use this to create a closed form algorithm 
for computing relative i?-polynomials. The uniform nature of this algorithm allows us to show 
combinatorial invariance of relative Kazhdan-Lusztig polynomials in the Hermitian symmetric 
space setting. 



1. Introduction 

1.1. Background. In their seminal work [11], Kazhdan and Lusztig defined for any Coxeter group 
W a class of polynomials indexed by W x W now known as the Kazhdan-Lusztig polynomials of W. 
These polynomials are usually defined in terms of an auxiliary set of polynomials, the i?-polynomials, 
also indexed by W x W. 

In 1987, Deodhar introduced parabolic analogs of these sets of polynomials in [7], which he 
referred to as the relative i?-polynomials and relative Kazhdan-Lusztig polynomials. The relative 
polynomials reduce to the standard ones in the case that a trivial parabolic subgroup of W is used. 
He provided a recursive calculation for these polynomials similar to the original formulations of 
Kazhdan and Lusztig. 

In P2], Proctor obtained the Hasse diagrams of quotient Bruhat orders that constituted lattices, 
and in [14], he gave a combinatorial game which again yielded these lattices, but with an explicit de- 
scription of the reduced decompositions of the elements of W/Wj associated to Hermitian symmetric 
pairs. In [10j . Jakobsen found the same diagrams as the lattices of noncompact positive roots, where 
he referred to these diagrams as being of Hermitian type, a practice we will continue in this paper. 
The diagrams, or at least their outlines, have since appeared in many different contexts, including 
[T2] and [H Section 3], where they are used to define trees from which relative Kazhdan-Lusztig 
polynomials can be derived, [4] , [5] , and [3] , where lattice paths of particular types are studied, and 
[16[ Sections 1.2 and 7] where they are used for calculations in quantum Schubert calculus, to name 
a few. 

Let q be a complex simple Lie algebra, p a maximal parabolic subalgebra with Levi decomposition 
p = i + u, where f! is the subalgebra whose basis is the root spaces spanned by compact roots, and 
u the subalgebra corresponding subalgebra for the span of the noncompact roots. It is well-known 
that the following conditions are equivalent: 

(1) (g,?) is a (complexified) Hermitian symmetric pair, 

(2) u is abelian, 

(3) the associated flag manifold G/P is cominiscule. 

This equivalence connects Jakobsen's work, which centered on the positive noncompact roots of 
simple complex Lie algebras, and Proctor's, which focused on the combinatorics of cominiscule flag 
varieties. In this work, if W is the Weyl group of G, and Wj the Weyl group of P, and G/P is 
cominiscule, we abuse notation and refer to the quotient W/Wj as a Hermitian symmetric pair. 
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1.2. This paper. The purpose of this work is to study the relative i?-polynomials for Hermitian 
symmetric pairs by means of a marking algorithm on diagrams of Hermitian type. In particular, 
for W/Wj a Hermitian symmetric pair, and u < v in W/Wj , we show that there are canonically 
obtained diagrams M for u and A for v such that M is contained within A, and the corners of 
the skew diagram A\M along its Northwest edges determine the relative i?-polynomial R^viq) 
completely. As an application of the algorithm, we prove that in this situation, v (q) is dependent 
only on the poset isomorphism class of [u, v]. In particular, if [u, v] is an interval in W/Wj and 
[u', v'] is an interval in W'/Wj', then R^ v (q) = R J U , v \q)- This settles the question of combinatorial 
invariance for the relative ^-polynomials for W/Wj a Hermitian symmetric pair in a uniform way. 
As relative Kazhdan-Lusztig polynomials are sums of relative i?-polynomials, their combinatorial 
invariance follows. 

It should be mentioned that a different combinatorial derivation of the i?-polynomials has been 
recently published by F. Brenti (for A n /A p _i x A n _ p , see [I], for all other types, see [5]). The 
algorithm we give has the benefit of being uniform for all types including E 6 /D 5 and Et/E§. 



2.1. Posets. We follow [T5] for all definitions involving posets. In particular, if P := (P, <) is a 
poset, u,v G P, we write [u,v] :— {w : u < w < v} and call this the interval of u and v. Similarly, 
(u, v) := [it, v]\{u,v}. For u G P, we refer to the lower interval of u to be the set {w : w < u}. If 
u < v, and (it, v) = 0, then we say that u is covered by v, and write u < v. We say that u and v are 
comparable if u < v or v < u holds. 
Recall that for m G N, 



where k = m if m is odd, and k = m — 1 if m is even. 

2.2. Coxeter groups. We follow [TJ Chapter 2] for all Coxeter group notation and terminology. 
In particular, given a Coxeter system (W, S), where S is a set of involutions generating W, we can 
define a rank function I : W — > N by £(w) = minimal number of generators required to write w. 
Define the left descent set Dl{W) := {s G S : £(sw) < £{w)}, and similarly the right descent set 
Dr{w). For v, w G W with v < w, let £(v,w) :— £(w) — £(v). Denote the identity of W by e . If 
JcS, let Wj, the parabolic subgroup of W generated by J, be the system ((J), J), and let 



We will identify W/Wj with its minimal right coset representatives for the remainder of this paper, 
and write W J for W/Wj. Then we have a well-defined length function £ w .j (wWj) :— £(w), where 
w is the minimal length right coset representative of W/Wj. This length function makes W J into a 
ranked chained poset [TJ Section 2.5]. All Coxeter groups appearing in this paper are crystallographic, 
and therefore are Weyl groups. 

2.3. Bruhat order. Weyl groups bear an important order, the (strong) Bruhat order, which can be 
realized in several ways. The general notion for covering in the strong Bruhat order is that u<Bruhat v 
provided £(u) < £(v) and there exists a reflection t = wsw -1 , s G S, such that tu — v. The strong 
Bruhat order is the transitive closure of these covering relations. An equivalent characterization of 
the strong Bruhat order is the subword order. 



2. Preliminaries 



[m] 




711—1 



W/Wj := {w G W : £(sw) > £(w) for all s G J}. 



Theorem 2.4. If u,v G W, then u < v if and only if there is a reduced decomposition of v such 
that a reduced decomposition for u can be obtained by deleting £(v) — £{u) simple reflections. 
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Proof. See [2 Theorem 2.2.2] 



□ 



A second order on Coxeter groups also exists, called the (left) weak Bruhat order. The covering 
relations are given by u Oi c ft weak v provided £(u) < £(v) and there exists s e S such that su = v, 
and the weak Bruhat order is the transitive closure of these covering relations. 

Both the weak and strong Bruhat orders descend to quotients of Coxeter groups by restricting 
to minimal coset representatives. It is well known that the strong and weak Bruhat order coincide 
provided W J is a Hermitian symmetric group, see [2] Section 2]. By abuse of notation, we use < 
for all Bruhat orders in this paper. 

2.5. Relative i?-polynomials. The object of study in this paper is a collection of polynomials 
introduced by Dcodhar in pj Lemma 2.8]. 

Theorem 2.6 (Deodhar). Let (W, S) be a Coxeter system, and J C S. For each x G { — 1, q], there 
is a unique set of polynomials {Ru%(q)}u,veW J C 1\q\ such that , for all u, v 6 W J , 

(1) Ri>Z(q)=Oifu£v, 



(2) R J u %(q) = 1, 

(3) if s G Dj,{v) and u < v 



' R J ' X (a) 

/>•;:>/: <; (q-l-x)Rt? sv (q) 

{q-l)Rl>Uq)+qR J s £ s 



if su < u 

if u < su W J 

if u < su G W J 



The polynomials from the above theorem are the parabolic (relative) i?-polynomials of W J of 
type x. If J = 0, it is trivial to verify that R^'~ 1 (g) = R%% = R u , v (q), where R u , v (q) is the standard 
i?-polynomial as defined in [U Theorem 5.1.1]. 

In this paper, we will be working only with the polynomials for which x = — 1. 



3. Essential definitions 
3.1. Diagrams of Hermitian type. 

Definition 3.2. We refer to the following diagrams as the diagrams of Hermitian type: 
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E 6 /D 5 
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Et/Eq 

Turning a diagram of any above type counterclockwise by 135° yields a lattice that is the Hasse 
diagram of the Bruhat order of the specified quotient. This fact was shown by Proctor, and the 
following is essentially a restatement of [13], Proposition 3.2. 

Proposition 3.3 (Proctor). The subdiagrams of W/Wj correspond bijectively to the elements of 
W/Wj , for each pair W/Wj listed above. 

Proctor provided a different proof of this result in [141 Theorem A] using a modification of a 
numbers game of Mozes. Alternatively, we could show the result using a sorting game that is 
essentially dual to Proctor's method. We give the details in A n jA v ^\ x A n ^ p in the following 
section. 

These Hasse diagrams appeared almost simultaneously in [TUJ Section 4] as the noncompact root 
lattices associated with unitarizable highest weight modules. 

3.4. Diagrams and partitions. The following definition is fundamental to this paper. 

Definition 3.5. A subdiagram of one of the above Hermitian type diagrams is a subset of the given 
diagram such that, if a box b is in the subdiagram, every box located above and left of b is also in 
the subdiagram. 

Importantly, diagrams retain the labels, which we think of as corresponding to simple reflections 
in the standard (Bourbaki) ordering of the generators S of the Weyl group W, imagining J C W 
as laying in S in the canonical way. 

Definition 3.6. A partition is a diagram with its labels removed. We refer to the partition A 
underlying a diagram A as the shape of A. 

Then the subdiagrams of the above can be interpreted in the following way: each w 6 W J corre- 
sponds to a partition of £(w) of type W J with the following properties: 



Quotient 


partition description 


A n /A.p— i x A n —p 


A = (Ai, . . . , Afe), k < n + 1 -p, Aj < p, 
where s p is the unique simple reflection of s not contained in J. 


C n /A n -i 


A = (Ai, . . . , A m ), n > Ai > A 2 • • • > A m > 0. 


B n /B n -i 


A = (to), to < n + 1, or A = (n + 1, 1, 1, . . . , 1). 




A = (Ai, . . . , A m ), n > Ai > A 2 • • • > X m > 0. 


A./A.-1: 


A = (to), to < n, or A = (n, to), to € {1, 2} or A = (n, 2,1,1,..., 1). 


Ee/D 5 ,E 7 /E 6 


irregular, best considered individually. 



Furthermore, because we require subdiagrams to be upper left justified, knowing a partition and its 
type is enough to construct the diagram for that partition. 
In this light, it makes sense to make the following definition: 
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Definition 3.7. The partition of a permutation w is the partition underlying the diagram associated 
to to through the bijcction detailed in the following section. 

The next definitions will only be used in Section [TOl but we state them now for completeness. 

Definition 3.8. We say that a partition is standard if it is a Young tableau under the usual 
definition (i.e. consists of a top and left justified set of rows, weakly decreasing in length). We say 
that a partition is standard skew if it can be written as a skew tableau of two standard tableaus. 
Note that all standard tableaus \i are standard skew, realized as /i\0. 

Definition 3.9. We say that W J supports a diagram of shape A//i if A\/i corresponds to A\M, 
with A the diagram of v in W J , and M the diagram of u. Given u < v in W J with diagrams M 
and A, respectively, we refer to A\M as a skew diagram of type W J , or of type W, when J is clear 
from context. The support for the skew diagram for v\u where u and v are specific elements of some 
quotient W J is the set of simple reflections that appear in the filling of v\u. 

4. The sorting game 

4.1. The idea of the algorithm. An alternate realization of the diagrams of Hermitian type can 
be had by modifying a sorting game developed by Eriksson and Eriksson [9, Section 9] . We begin by 
noting that each Weyl group can be associated to a permutation group: A n with the symmetric group 
on n+1 letters, etc. We realize each of the above quotients in their permutation representation, using 
the permutation representation described in Section 6.2] for E^/D^ and Ei/E§ and the standard 
representations described in pQ Chapter 8] for the others, then use a sorting argument to construct 
tableau. The rows in the constructed tableau correspond to rows in the above diagrams in a natural 
way, and because we are working in Hermitian symmetric spaces, the Bruhat and weak Bruhat 
orders coincide on these quotients, allowing us to read the left descent sets of each element from the 
diagram (see Proposition [5?3|). Furthermore, because the sorting game is modeling multiplication in 
the Weyl group, the diagrams we derive can be associated to reduced decompositions. This will be 
a central tool in the statement and proof of the main theorem. 

We present a single case of the sorting game. The rest are very similar. 

4.2. Case: A n jA v -\ x A n ^ p . Suppose W — A n with the standard generating set S of adjacent 
transpositions, and J = S\{s p }. Then W J consists of permutations increasing in positions 1 through 
p (called segment one), and in positions p + 1 through n + 1 (called segment two). Write w(j) for 
the image of w under the action of the permutation w; if w is written in complete notation, we see 
this is just the j th entry in w. Define a game on w £ W J in the following way: in step one, sort 
w(p+ 1) into the segment [w(l), . . . , w(p)], and record the number of elements "jumped" as d\. Call 
the resulting permutation iv\, and sort w\{p + 2) = w(p + 2) into the segment [wi(l), . . . , u>i(p + 1)], 
again recording the number of jumped elements as c?2- as w(p + 1) < w(p + 2) by assumption, 
d-2 < d\. Continue until the permutation is completely sorted, leaving a partition (d±,d2, . . . ,dk). 
Note that 0<fc<n+l — p, and that k — if and only if w was sorted before any action was taken 
(which implies that w is the identity). 

Each sort can be realized as multiplication by a specific product of simple reflections; in particular, 
the action of sorting position p + k + 1 to position p — j is s p+ kSp+k~i • ■ ■ > corresponding to the 
diagram row word 



Sp+k 




Sp-j+l 


Sp-j 



Thus we can associate to each partition an element of W J by 

to = r(p, A fc ) _1 r(p, Afc-i) -1 • ■ ■ r(p, Ai) 
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and so we have a map 



<t> : A n /(A p -i X A n - P ) i— >• {A = (Ai,...,A fe ),|A| = £{w) 7 X 1 <p 7 k<n + l-p} 



where the labeling is exactly as in Section [3.21 This map is easily shown to be bijective. 



Remark. All further cases are similar, and so we give only the map 



<f> : W J m> 



{partitions with particular conditions}. 



4.3. C n /A n -i. Here, W can be viewed as Weyl group of type C n , i.e. permutations of {— n, . . . , —1, 1, 
with w(i) = —w(—i), and where we take J = A n _i. We see that W J := {w S C n : 3!1 < p < 
n with w(p) > 0, w(p) < 0, and w(l) < ■■ ■ < w(p),w(p + 1) < • • • w(n) < 0}. Note that w(n) > if 
and only if w is the identity in this case. The algorithm is as follows: set w'(n) = —w(ri), and sort 
it into [iu(l), w(2), . . . , w(n — 1)], recording d\ to be one more than the number of jumped elements, 
and call the resulting permutation w\. Continue until Wk(n) > 0. Then (f>(w) — (di, d,2, . . . , dk)- As 
we assumed that w(i) < w(J) < 0, and w(i) is sorted before w(J), then w(i) is moved more positions 
to the left than showing that dj < di for j > i. Thus we have a map 



Using a row word argument with the labels fixed as before, this map is bijective. 

4.4. B n /B n _±: Take W = B n , and J the standard generating set of -B„_i chosen such that it lies 
within the standard generating set for B n , with the same enumeration. Recall that as permutation 
groups, B n and C n are isomorphic, so w £ B n is a permutation of [— n, ■ • ■ , — 1, 1, • • • , n] such that 
w(i) — —w(—i) for 1 < i < n. Thus we can write w as [u>(l), . . . ,w(n)], and W J = {w € B n : < 
w(2) < • • • < w(n)}. The only negative element, if there is one, is in position 1. The sort has two 
parts: first, if w(l) > 0, sort w(l) into the segment {w(2), . . . , w(n)}, recording the number of right 
jumps as di, and the algorithm terminates. If w(l) < 0, let W\ be (w(2), . . . , w(n), —w(l j), recording 
d\ = n. Next, if necessary, sort {wi(n — l),u>i(n)}, recording the number of left jumps of wi(2) as 
c?2- Continue sorting larger terminal segments in increments of one; note that as we increase the size, 
we can at most move an entry one position, thus dk S {0, 1} for k > 1. The algorithm terminates 
when dk — 0. 

4.5. D n /A n _±. Here W is Weyl group of type D n , i.e. permutations of — [n] U [n] with an even 
number of negative entries in positions 1 through n, and with w{i) = —w(—i) for 1 < i < n. 
Again we write w 6 D n as [w(l), . . . ,w(n)]. It is easy to see that W J :— {w G D n : 3!1 < p < 
n with w(p) > 0, w(p) < 0, and w(l) < ■ ■ ■ < w(p), w(p + 1) < • • • w(n) < 0}. Given a permutation 
w , we have a two-step method of sorting: first, let w'(n) = —w(n — 1), and w'(n — 1) = — w(n), 
and begin by sorting w'(n — 1) into {w'(l), w'(2), u>(3), . . . ,w(n)}, noting d\ as one more than the 
number of jumped elements, and denote the resulting permutation as w" (note that w" ^ W J ). 
Then sort w'(n) into {w" (1), w"(2) ■ ■ ■ ,w"(n — 1)}, letting d^ be the number of jumped elements. 
Again, by our assumptions d-i < d\ (as w'(n) < w'(n — 1), implying it is sorted fewer steps right). 
Continue the same process with the resulting permutation. Note that dik may be zero for some k, 
and this may happen only if there are no remaining negative entries. The process halts when no 
negative entries remain at the beginning of an odd step. This gives a map 



: C n /A 



n-1 !-> 



{A = (Ai, . . . , A fe ), |A| = £(W), Xi < Xj if i < j, Ai < n} 



<f> : DJA 



n-1 !-> 



{A= (Ai,...,A fe ) : |A| =£(w),\i < Xj ift<j,Ai <n-l}. 
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4.6. D n /D n -i. In this case, the Bourbaki ordering of simple reflections admits no "nice" combinato- 
rial interpretation in terms of the game we have presented so far. Instead, we present the proof using 
the ordering given in [TJ. Explicitly, if we denote the Bjorner and Brenti enumeration of generators 
bi, < i < n — 1, and the Bourbaki ordering Si, 1 < i < n, we have that bi = Sj for 1 < i < n — 1, 
and 6o acts as the cycles (—1, 2)(1, —2), where s n acts as the cycles (n — 1, — n)(n, — n + 1). This is 
effectively a diagram bijection carrying Sj <-> s n _$. The difference is purely cosmetic, but it allows 
an easier statement of the sorting algorithm. 

Take W = D n , and J the generating set of D n _i chosen such that it lies within the standard 
generating set for D n) with the same enumeration. Then W J = {w G D n : w(—2) < w(l) < ■ ■ ■ < 
w(n — 1)}. Again, the sorting process has several steps: begin by sorting w(n) into {w(l), . . . , w(n — 
1)}. We observe that this step is trivial only in the case w — e, as otherwise w(n) < w(n — 1) 
as is easily verified from the definition of the quotient. Explicitly, w(— 2) < w(l) < w(2) implies 
that if any negatives appear in positions 1 through n of w , than they must be in positions 1 and 
n , implying that w(n) < < w(n — 1); if no negatives appear, then o\ = {w(l), . . . ,w(n — 1)} 
represents an ascending sequence taken from [n], and it is only the trivial sequence that fails to 
utilize n in a\, showing w(n) < n = w(n — 1). Record the number of elements jumped by w(n) as 
it moves left as d\. 

If there are no negative entries, then the resulting sequence is sorted, and the process terminates. 
Otherwise, denote the resulting permutation as u>i, and let w'(l) = — wi(2), w'(2) = — W\{1). 

Sort Wi(3) into the segment {w'(l), u>'(2)} recording one more than the number of jumped ele- 
ments as d,2 (note that it is possible that no jump occurs). We claim that 1 < < 2: to see this, note 
first that w(— 2) < w(l) < w(2) implies w(— 2) < w(— 1) = —w(l) < w(2), and also that, as we must 
assume w(n) < to have reached this step, we know that the sets {w'(l), w'(2)} = {—w(l), — w(n)}. 
Hence sorting u>(2) into {w'(l), w'(2)}, we can't pass —w(l), meaning we can jump at most one 
element. 

Denote the resulting permutation as u>2, and jump u>2(4) into {u>2(l), it?2(2), W2(3)}, recording 
as the number of left jumps; again, and for the same reason, d^ < 0. Continue until no left jump is 
possible, at which point in time the sequence is sorted. The partition is [d\ 1 ■ ■ ■ , dk), and labeling 
as in the figure realize the sorts involved, giving the bijection as before. 

4.7. Eq/D§ and E^/Eq. These cases are best checked by hand, using the permutation descriptions 
Remark. The row words that appear in each case give the labeling of the diagrams in 13.21 

5. Order and descents 

5.1. Young's lattice. Recall Young's lattice on partitions: two partitions ji = (pi, . . . , /i r ) and 
A = (Ai, . . . , A s ) are related by [i < A provided r < s, and that for 1 < i < r /ij < A^. Visually, if the 
partitions are represented as Young diagrams, this just says that the diagram of fi sits inside that 
of A when their upper left corners coincide. 

In the current context, if M and A are diagrams of type W J with shapes /i := (/ii, . . . , /z r ) and 
A := (Ai, . . . , A s ) respectively, then the labels in their upper left corners again coincide, and we again 
say that M <diagram A provided /ii < Ai for each 1 < i < r < s. We call this the containment order 
on W J , or just the containment order when the context is clear. It is a consequence of the definition 
of subdiagrams that W J is a poset with respect to this ordering. 

5.2. Diagrams and order. Given a diagram A of type W J , we can visually distinguish the ascents 
and descents of w € W J associated to A. 
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Proposition 5.3. Let u G W J with diagram A, and suppose that su G W J with diagram M . Then 
s G Dl{u) if and only if the skew diagram A\M consists of a single box, positioned in an outside 
corner of A. Similarly, if s ^ Dl(u), then M\A is a single box that occupies an outside corner of 
M. 

Proof. If s is a left descent of u G W , then as the weak Bruhat and strong Bruhat orders coincide 
for Hermitian symmetric pairs, we can write any reduced decomposition of u as ash, with sa = as, 
with s not appearing in a. Then in the lexicographically minimal reduced form, we can represent 
u as ash as above, and a can contain no copy of s, and as = sa. but this says no s' labeled box 
appears in a, which means no s' labeled box appear in the partition for u after the final occurrence 
of an s-labeled box, i.e. that s-labeled box is an outside corner of A. 

Conversely, if s is the label of an outside corner of A, say on row j, then s commutes with all 
labels of boxes that appear on rows lower than j, and nothing appears right of the last s-labeled 
box on row j. Thus s commutes with all simple reflections that appear left of the first occurrence 
of s in the lexicographically minimal reduced decomposition, implying s is a left descent. 

The second statement is similar. □ 

5.4. A poset isomorphism. The following follows easily from Proposition [5T3] 

Corollary 5.5. The poset (W J , <diagram) is poset isomorphic to (W J ,<i e ft weak)- Thus u < v in 
W J if and only if the diagram of u is contained in the diagram of v. 

Proof. Covering relations in the left weak order are determined precisely by left simple descents. By 
the above, s is a left descent of v if and only if an s-labeled box is an outside corner in the diagram 
of v. 

If u covers v in the left weak order, then there exists s G Dl(v) such that su — v. By Proposition 
15.31 there is an s-labelled outside corner in the diagram A of v that is not contained in the diagram 
M of u. Then the s-labeled box can be removed from A, leaving a subdiagram, and considering 
the diagrams as products of row words forming a reduced decomposition, this corresponds to left 
multiplication by s. Thus the resulting diagram is M. 

Similarly, if M C A, with M the diagram of u and A the diagram of v, then this presentation 
by diagrams witnesses that it is a subword of v, and so u < v by the Subword Criterion (Theorem 
l2~3 . □ 



6. The main theorem 

6.1. Marking the diagrams. Given u < v in W J , let A be the diagram of v, and M the diagram 
of u. As u < v, then M G A by Corollary 15.51 By Proposition 15. 3[ s G Dl(u) if and only if an 
s-labeled box is an outside corner of the diagram of u, and su is a minimal coset representative 
covering u if and only if we can append an s-labeled box to M in such a way that the resulting 
diagram is a subdiagram in the sense of Definition 13.51 
For an s-labeled box in A\M , define 

6(b) := ^{s-labelcd boxes in A\M above and left of b, including b}. 

We define a box b to be even provided 5(b) is even, and similarly for odd. 

(1) If s ^ Dl(u), su G W J , and s is a short (long) reflection, then mark every (odd) s-labeled 
box with a +. 

(2) If s G Dl(u), and s is a short (long) reflection, then mark every (even) s-labeled box in 
A\M with a -. 
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For each box b in A\M, define 
A(6): 



#{boxes in A\M above and left of b , including b} if b marked 
f if otherwise 



Suppose u < v, and that the diagram of v is A, and the diagram of u is M. Then we define 
k u ,v '■= #{+ marked boxes in A\M} — marked boxes}. 



6.2. Statement of Main theorem. Now we can state the main result of this paper. 

Theorem 6.3 (Main Theorem). Given u < v in W J of rank n, let A be the diagram of v, and M 
the diagram for u, and mark the skew diagram A\M as in Section^ Define 



(6.3.1) 




if u = v 
if u ^ v 

(q-l) k ^Il beAW [±(b)] a ifu<v, 



where the second product should be interpreted as a product over all boxes in the marked diagram 
A/M, and a is +1 for + marked boxes, —1 for — marked boxes, with r/(u,v) is the unique natural 
number so that the degree of the right hand side is £(v) — £(u). Then v (q) = R^ v (q). 

For ease of discussion, let f^ v (q) be the function defined as R^ v {q), but with n u ^,(s) = for all 
s. We provide several examples to aid in the discussion: 



6.4. Examples. 

Example. Cons 
Again, we use the convention of using i for Si . Then 



Example. Consider u = ((543)(65)) \ v = ((54321)(6543)(7654)(8765)(9)) 1 in A w /A 4 x A 5 . 



A\M 



5 


4 


3 


6 


5 





5 


4 


3 


2 


1 


6 


5 


4 


3 




7 


6 


5 


4 




8 


7 


6 


5 




9 









X 


X 


X 






X 


X 


X 


+ 




X 


X 








X 


X 


+ 














+ 






+ 












+ 



















(,-i)V°(i + ?). 
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Example. Let u = ((86543)(7)) 1 and v = ((8654321)(765432)(86543)(7654)(865)(7)) 1 in D s /A 7 . 
Then we have 



8 


6 


5 


4 


3 




7 





8 


6 


5 


4 


3 


2 


1 




7 


6 


5 


4 


3 


2 




8 


6 


5 


4 


3 






7 


6 


5 


4 








8 


6 


5 










7 





A\M 



X 


X 


X 


X 


X 








X 

















































X 


X 


X 


X 


X 


+ 






X 


+ 








+ 






+ 












+ 












+ 













and ^(g) = (q- l^J^H^ie = {q _ 1)2(1 + q + ?2)? 16. 

Example. Take u = ((13456)(245)) _1 and v = ((13456)(245)(342)(1345)) _1 in E 6 /D 5 . Then 



1 


3 


4 


5 


6 




2 


4 


5 



1 3 


4 


5 


6 








2 


4 


5 










3 


4 


2 








1 


3 


4 


5 



A\M 



X X 


X 


X 


X 




X X 


X 


X 


X 






X 


X 


X 




X 


X 


X 




















+ 




















+ 







and R u , v (q) = (g - I) 2 " 1 M q 6 = (? - l)? 6 - 

Example. Let u = ((13456)(2)) _1 and w = ((13456)(245)(342)(13456))" 1 in E 6 /D 5 . Then 



1 


3 


4 


5 


6 




2 





1 3 


4 


5 


6 










2 


4 


5 












3 


4 


2 










1 


3 


4 


5 


6 



A\M 





X 


X 


X 




X X 


X 


X 


X 






X 








X 


+ 
























+ 




















+ 





with R u ,v{q) = (g — 1) 



3-2 



[3! 



[2] [3] 



<7 9 = (g - !)9 9 - 



7. Strategy of the proof, and a trivial case 



7.1. Strategy. We will show that v (q) obeys the recursions required by the relative i?-polynomials. 
One case of the recursion is trivial, and will be treated case independently in Proposition 17.31 The 
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strategy for the proof of the other recursive rules is as follows: we identify the possible diagrammatic 
presentations of each rule, and reduce to a trivial identity about quantum integers. The discussion 
in Section [5] will detail how we can restrict the skew diagram associated to the pair u < v to a 
smaller diagram and prove the recursion there. 

7.2. A simple case of the recurrence. We can show that the i?-polynomials satisfy the sec- 
ond recursive relation in Deodhar's definition of relative i?-polynomials without appealing to any 
particular Hermitian symmetric pair. 

Proposition 7.3. Let u, v £ W J with u < v, and suppose that s £ D l(v)\D l{u) , and that su W J . 
ThenRl v (q)=qRl sv (q). 

Proof. If A and M are the diagrams for v and u respectively, the situation in the proposition is that 
there is an s-labeled outside corner of v that is neither an inside or outside corner of u. Thus the 
s-labeled boxes in A\M are not marked with + or — . Similarly, if A' is the diagram for sv, the only 
difference between A and A' is a single unmarked box, implying that r J u v (q) = r J u sv (q) . Then the 
only difference between R^ v (q) and qR^ sv (q) is a single factor of q, yielding the result. □ 

8. The case s £ D L (v) n D L (u) 

8.1. A remark on markings. Suppose that s £ Dl(v), and that u < v. Let A be the diagram of 
v, with A' the diagram for sv, and M the diagram for u, with M' the diagram for su if su £ W J . 
Suppose furthermore that W J is not Eq/D^ or Eg. 

If s £ Dl(u), then there is an s-labeled outer corner in M. Then the skew diagram A\M differs 
from A'\M' in the diagonal containing that outside corner, and possibly in the diagonals immediately 
above and below this diagonal. The upper diagonal differs if s'u > u and s'u £ W J , where s' is the 
label of the box immediately right of the s-labeled outside corner in the diagram for u, and a similar 
condition holds for the lower diagonal. 

In almost the same way, if s € Dl(v)\Dl(u), and su £ W J , the diagrams A\M, A'\M, and 
A'\M' differ at most on the diagonal of A\M containing the s-labeled outside corner, and on the 
diagonals immediately above and below it. 

Thus the difference in the polynomials R^ v (q) , R^ sv (q) , and R J SU sv (q) is determined by markings 
that differ on at most three diagonals. We refer to this set of diagonals as /, and define R^ v (q)\j to 
be the polynomial determined by the markings on the diagonals in /. 

8.2. Another case of the recurrence. With this remark in hand, we can take on the next case 
of the recursion for R^ v (q). 

Proposition 8.3. Suppose that s £ D L (u) n D L {v). Then R^ v (q) = Ri u>sv (q)- 

Proof. In this case, su < u, implying su £ W J , so in particular the diagram for su is contained 
within the diagram of u, and is in fact obtained by deleting the s-labeled outside corner. Let A v , 
M u , and M su are the diagrams associated to v,u, and su, respectively. Analyzing the diagrams 
of Section 13 . 21 we see the following marked diagrams. Note that we draw only the three relevant 
diagonals, and do not cover the cases E 6 /D 5 or Et/E 6 , or include diagrams symmetric to given 
diagrams. In each pair of diagrams, the left one is taken from A\M, and the right one is from 
A'\M', and it is a matter of definitions to see that f„ v (q)\j = fg U SV (q)\i. 
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To be explicit, consider the restricted diagram marked as below: 



X 


X 






X 


X 








X 


X 








X 


+ 






















+ 
















































+ 



























Assume that the length of the — marked diagonal in the first diagram is k; then f^ v (q) 
1 (q-l) k M 



(q - l) k [k\Y r ™>™(<l) ((? _ 1) 2/ S [ fc! ][ fe! 



, showing the two are equal. As the skew diagram 

rj(su,sv) 



for the pair su < sv contains exactly as many boxes as the diagram for u < v, q^ u ^ = q 

showing Ri tV {q) = R J SUtSV {q) in this case - 

All other cases are similar; the relevant diagrams appear below: 



X 


X 


X 




X 


X 


+ 










+ 


























+ 









X 


X 


X 




X 


+ 












+ 


























+ 













X 


X 


X 






X 


X 


+ 






X 


+ 




+ 










+ 




+ 
























+ 




+ 










+ 





X 


X 


X 






X 


+ 








X 




+ 












+ 




















+ 













\x\x\ 





\x 


X 


+ 




\x 


+ 










+ 




+ 








+ 






x\+\ |... 










+ 






+ 
































+ 



















\x 


X 


X 






\x 


X 


+ 






\x 


+ 








X 


X 










+ 




+ 






+ 








+ 














+ 








+ 




















































+ 










+ 





\x 


X 


X 








X 


X 










+ 









































\x 


X 


X 








X 


+ 










+ 










+ 

























\x 


X 


X 








X 


+ 










+ 










+ 

























III 



X\ + \ |... 

















By the discussion above, this shows that if u 

KM = RksM fOT WJ i {E 6 /D 5 ,E 7 /E 6 }. 
using the computer. 



< v in W J , and s G D L (v) D D L (u), then 
The cases E & /D 5 and E-j/Eq can be verified 

□ 
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9. The case u < su, su G W 



j 



9.1. Strategy, and a reduction. As noted above, this proof will proceed by noting the possible 
restricted diagrams that may appear with an s-labeled outside corner of v, and an s labeled inside 
corner of u. In each case, we provide the diagrams associated to u < v, u < sv, and su < sv, 
respectively, and reference the part of Lemma f9T3l (below) that is used in showing that R'u yV {q)\i = 

(Q-^)Ksv(Q)\l + A,sv(Q)\l- 

Let A x denote the diagram of x in W J , and d UyV (I) the number of boxes in the restriction to / 
of A„\A U . Then d UtV (I) = d UtSV (I) + 1 = d SUtSV (I) + 2. Note that R J UyV {q)\i = rt, v (q)q 9 , where g is 
d UjV (I)-deg(r^ v (q)\i). In particular, if we wish to show that R J u ^{q)\i = {q- 1 )R'Lsv(q)+qR'L,sv(q), 
we can instead show r^ v (q)\i = q a (q - l)f^ sv {q)\i + q b ■ qf^ UiSV (q)\i, where o + deg(f^ >OTJ (g)|j) + 1 = 
deg(f„ v (q)\i), and similarly for b. Then multiplication by q 9 , where g is as above, generates the 
desired identity. Calculating g introduces extra notation, and can be avoided in the proof that 
follows. Instead, we will multiply the terms coming from the diagrams for u < sv and su < sv by 
the power of q to bring those terms to degree d UiV (I) — 1, resp. d UiV (I) — 2. 

9.2. A simple lemma. We require a lemma to make the proof. 
Lemma 9.3. For and k > 1, 



(1) [fc] = [fc - 1] 

(2) [k] = l + q[k 



„fc-i 



Proof. The result follows from the definition of quantum integers. 



□ 



9.4. The final case of the recurrence. To finish the proof of the main theorem, consider first 
the restricted diagram 



X 


X 


X 






X 


X 


+ 














+ 




























+ 














+ 



X 


X 


X 






X 


X 


+ 












+ 


























+ 













X 


X 


X 






X 


X 


X 





















































with outside corners north of and west of the inside s-labeled corner, occurs in the Type A n quotients, 
as well as C n /A n -x, D n /A n -i, Eq/D^, and Et/Eq, with fc + marked boxes. This diagram yields 

2-k [k] 



which is of degree 1 

(fc-2)(fc-l) 



K,M\i = {q- i) 



■ Then K,sv - 



[(fe-iK 

{ q -iy- k 



[(fc-1)!] 



which has degree 1 



_ ( K ^)\ K a ^ differs from expectations by 1 — fc, and f J su sv (q) = (q — l) 2 k ( k l 2 )\ w ^ n degree 
2 — fc — ( fc 3 H fc 2 ) ; differs from the expected degree by 1. Thus the relevant identity to show is 



2-k 



[fc] 



[(fc-1)!] 



= (9-1)9 
(q - lf- k 



[(fc-1)!] 



qq 



2-k 



(fc-2)! 



(q 



fc-i 



[fc - 1]) 



[(fc-1)!] 
which is Lemma l9~3l (1). 

From here, we describe on, we cease to explicitly calculate the expected degrees. In all cases, fc 
is the number of + signs in the first diagram of a triple. 
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For our next diagram, take 



X 


X 


X 




X 


X 


X 




X 


X 


X 






X 


+ 








X 


+ 






X 


X 


+ 










+ 










+ 










+ 




































+ 













corresponds to the identity 

(q - l)[fe] = (q- l)q k ~ l + q ■ q-\q - l)[fe - 1] 

which reduces to Lemma l9~3l (1). 

This appears in the Type A n quotients, as well as C„/A n -i, D n /A n -i, Eq/D^, and Et/Eq. 
Next consider 



X 


X 


X 




X 


X 


X 




X 


X 


X 




X 


+ 






X 


+ 






X 


X 


+ 








+ 










+ 








+ 




+ 


































+ 














+ 





corresponds to the identity 

(q - l) k [kl] =(q- l)[(fc - l)l]q k ~ l + q(q - \) k [k - l][(fe - l)!]^ 1 

which reduces to Lemma l9~3l (1). 

This appears in the Type A n quotients, as well as C„/A„_i, D n /A n -i, E^/D^, and Ej/Eq. 

We now move to restricted diagrams that appear in D n /A n -i and C n /A n -i. Consider the 
restricted diagram of A\M 



X 


X 


X 






a 


c 










b 


c 










a 


c 
















a 



where a is s„, and b is s„_i, or vice versa. Then we obtain the following triple of marked diagrams 



X 



X 



\x 


X 








+ 




















+ 

























\x 


X 








X 


+ 










+ 










+ 




















+\ 



(note the first box in the bottom row of the last diagram cannot be marked, by reason of parity). 
Suppose there are k + marked boxes in the first diagram. Then this restricted diagram corresponds 
to the identity 



(q - l) fe [l][3] ••• [2k - 1] = (q - l)V fc ^[l][3] • ■ • [2k - 3] + (q - l) fe [l][3] ■ • ■ [2k - 3][2k - 2] 



which reduces to Lemma T9.31 (1). 
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The restricted diagram of A\M 



X 


X 


X 






a 


b 










a 


b 










a 


b 
















a 



with a — s n , b = s n _i with an odd number of copies of s n appears as a restricted interval in 
C n /C n _i, and by our labeling algorithm, bears the same marks as the above case, and so satisfies 
the same identity. 

The same restricted diagram, but with an even number of s n labeled boxes, appears in C n /C n -i 
but not D n /D n -i (as we assume in that the label in the lowest box b is a descent of v, which in D n 
requires b to be an odd box). Here the identity is 

(q l) fc [l][3] • • • [2k - 1] = q- 1 ((q l) k+1 [l}[3] ■ ■ ■ [2k - 1]) + q^ (q(q l) fc [l][3] ■ ■ ■ [2k - 1]) 

which is true by inspection. 

Consider the diagram A\M corresponding to u < v 



X 




X 


k 




n-2 


n 




n — 1 


n-2 






k 



with marked triple 



|X|...|A1 + | |... 














+ | |... 






|X|...|X|JT| + |... 










+ 


























+ 



and note if two boxes are + marked, their weights are 1 and k, where the + marked box is labeled 
Sfe. From this, we can read R'l v {q) = (q — l) 2 [fe]q' £_1 , while R^ v (q) for the second diagram is 

R^ sv (q) = (q - l)<? 2fc ~\ and the third is Rs U , sv {q) = (<Z - l) 2 [fe - ±]q k ~ 2 ■ Substituting into the 
recursion, we can reduce to the identity in Lemma 1^731 (2). 
This diagram appears in D n / D n _i, and can be renumbered 



to be found in E e /D 5 and E 7 /E 6 . 

In Bn/Bn-i, the only relevant diagrams have the following shape 



X ...\X k\ 



k 
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with marked triple 



\X ... \X \ + \ 



\x\...\x\ 



\x\ ... \x\x\ 



We show the full diagram, but this is only to aid comprehension. The usual three diagonals are all 
that is important to the calculation. 
The relevant identity is 



(q - l)q k = (q-l).(q- l)q k ^ + q ■ (q - l)q k ~ 2 

which is trivially verified. 

As an aside, it is straightforward to use the above identity, and induction, to show the following. 

Proposition 9.5. If[u,v] is an interval in B n /B n -i, then R^ v (q) = (q — l)q £ (' u )~^(' lI )~ 1 . 

This type of interval appears in B n /B n -i, C n /C n ~i, Ee/D§, and E7/ E§. 

The remaining cases are similar, and we give only the diagrams which have yet to appear: 

In C n /C„_i and D n /D n _i we have 



X 


X 








X 


X 






X 


X 






X 


+ 








X 


+ 






X 


X 


+ 








+ 












+ 








+ 




+ 








+ 










+ 












+ 
































c 


+ 






c 











where e is — if there are an even number of + signs in the first diagram, and is unmarked otherwise, 
and £ is + if there are an odd number of + signs in the first diagram, and unmarked otherwise. 
Similarly, in C„/C n -i and _D„/_D„_i, we have the restricted diagrams 



\x 


x 


X 






\X 


X 


X 








X 


+ 








X 


+ 










+ 












+ 










+ 












+ 


































e 


+ 










c 





\x 


X 


X 








X 


X 


+ 








+ 




+ 












+ 

























with e and £ as above. 

Finally, there are finitely many cases arising in Eq / D§ and E7 / Eq that may be obtained by hand 
or computer program. 



10. Combinatorial Invariance 

10.1. The simplest cases. As an application of the above algorithm, we show that the intersection 
cohomology of the cominiscule flag varieties is a combinatorial invariant, i.e. the cohomology is a 
function only of the poset. Begin by recalling Proposition [5T31 which provides a characterization of 
all covering relations in the Weyl quotient posets of Hermitian symmetric type. In particular, the 
poset W J is isomorphic to the containment order on partitions of the given types. All that remains 
is to show is that a skew diagram uniquely determines an R J polynomial. 

Proposition 10.2. If a skew diagram v\u consists of a single row of length k, it can be identified 
with the polynomial (q — l)q k ~ 1 . The same statement is true for columns. 
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Proof. The leftmost box in v\u is easily observed to be an ascent of u within v for any diagram of 
Hcrmitian type. No other box is an ascent, or descent, of u, thus the polynomial is as stated. The 
proof for columns is almost the same. □ 

Next, we note that if two restricted diagrams from Type A have the same markings, then they 
have the same ^-polynomials. 

Proposition 10.3. If u < v in Type A, with shapes A and \i for v and u respectively, then R^ v (q) 
depends only on the skew shape A\/x. 

Proof. In the proof of the main theorem for Type A above, only the length of certain diagonals was 
used to construct R^ v (q), and the diagonals are determined uniquely by the skew shape v\u. □ 

10.4. Combinatorial invariance. Recall that we say that a diagram without its labels is a parti- 
tion. In particular, note that a diagram A is associated with a class of partitions, notably all such 
partitions that can contain that shape with that filling. Meanwhile the partition A associated A is 
associated with that class, and any other diagram M that, when stripped of its filling, has shape A. 
We will use upper case Greek letters for diagrams, and lower case Greek letters for partitions in this 
section. Recall in particular the definitions of Section [3] 

Proposition 10.5. If A\M is a standard skew diagram of type W J , it can be identified with a skew 
diagram in type A, and the R-polynomial associated to A\M in W J is the same as the R-polynomial 
of the same shape in type A. 

Proof. It is easy to see in the diagrams of Section 13.21 that a standard tableau of any type other 
than C„/j4„_i is supported on a set of simple roots whose Coxeter graph is an induced subgraph of 
a Coxeter graph of type A. 

For the C n /A n -i case, the standard skew diagram for v\u can include a single box labeled s n in 
the bottom left corner (see figure below) . If this corner is not an inside corner, then the box North 
and West of it (in the diagram for u) is an outside corner, and by the rules established for C n /A n _i, 
the box associated to s n is unmarked, as it would be in type A. If it is an inside corner, it is marked 
with a plus, exactly as it would be in Type A. All other boxes in the diagram are marked according 
to type A rules, and thus the polynomial associated to A\M in the context of the quotient C n /A n -i 
is the same as the polynomial associated with the same diagram in type A. □ 



\x 


3 


2 


1 


\x 


+ 






\x 


X 


2 


1 


\x 


X 


+ 






4 


3 


2 






+ 






4 


3 


2 




+ 




+ 



C n /A n -i standard, s n not inside 



C n /A n -i standard, s n inside 



We call a partition a staircase if it can be identified with the partition underlying a Hermitian 
diagram of type D n /A n —i. Note that the full list of maximal staircase partitions not explicitly 
belonging to D n /A n -i or C„/A„_i are 



n — 1 


n 




n-1 



6 B n /Bn 



e E 6 /D b or E 7 /E 6 



E E 7 /E e . 



But in all cases, the rules are those of D n /A n —i, and so we can identify polynomials again as before. 
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The final general form of skew partition is an L shape, of which there are two exceptional maximal 
examples, and one infinite class: 



n - 1 


n 




n- 1 




1 



£ B n /B 



n-l 



3 


4 




5 




6 




7 



E Et/Eq 



7 


6 


5 


4 




2 



G £ 7 /£ 6 



2 


4 




5 




6 




7 



E E^/Eq and restricted in E 6 /D 5 



In type B n /B n _i, the have R'^ v (q) = (q — l)^( u )~ £ ( u )^ 1 via Proposition 19. 51 while the rule in type 
E is clear: one plus in the upper left box, and no other signs. Thus the relative i?-polynomials are 
the same. 

The final cases to compare are intervals contained solely in Eq/D^ and Et/Eq, which requires a 
computer check. Thus we have: 

Theorem 10.6. If X\fi is any skew partition, and A\M, A'\M' are skew diagrams of shape 
from types W J and (W') J respectively, then R^ v (q) = R^, v >(q). 
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